Block search engine crawler

What is block search engine crawler

Usually, we redirect similar or sub domain names (301) to maintain content uniqueness, which is beneficial for optimizing SEO (search engine) ranking optimization. In order to better optimize search engines, we need to maintain content consistency to prevent websites with the same content from spreading our traffic. Block search engine crawler, also known as bots, index CDN content to prevent crawling programs.

How does block search engine crawler work

When the block search engine crawler is enabled, robots.txt will automatically display on the CDN URL, for example, http (s)://www.example.com/robots.txt, which includes:

User-agent:*

Disallow:/

This ensures that all search engine robots are prevented from indexing your CDN content.

How to configure block search engine crawler

You can choose to enable Block Search Engine Crawler in CDN Resources ->Safety Protection ->Access Protection, without the need to configure other content.